An intelligent video surveillance (IVS) system may be able to detect, track and classify objects in real-time. If the actions of an object are suspicious (i.e., deviate from normal behavior or violate one or more prescribed rules), the IVS system may send real-time alerts. The performance of the IVS system may primarily be measured by the effectiveness of these alerts, namely the detection rate and the false alarm rate. One factor affecting these two rates may include the quality of detecting and classifying the objects.